Delegating Network Processor Operations to Star Topology Serial Bus Interfaces

ABSTRACT

An advanced processor comprises a plurality of multithreaded processor cores each having a data cache and instruction cache. A data switch interconnect is coupled to each of the processor cores and configured to pass information among the processor cores. A messaging network is coupled to each of the processor cores and a plurality of communication ports. The data switch interconnect is coupled to each of the processor cores by its respective data cache, and the messaging network is coupled to each of the processor cores by its respective message station. In one aspect of an embodiment of the invention, the messaging network connects to a high-bandwidth star-topology serial bus such as a PCI express (PCIe) interface capable of supporting multiple high-bandwidth PCIe lanes. Advantages of the invention include the ability to provide high bandwidth communications between computer systems and memory in an efficient and cost-effective manner.

RELATED APPLICATIONS

This application is a continuation of Ser. No. 13/253,044 filed Oct. 4,2011, which is a continuation of Ser. No. 11/831,887 filed Jul. 31,2007, which is a continuation in part of Ser. No. 10/930,937 filed Aug.31, 2004, which is a continuation in part of Ser. No. 10/898,008 filedJul. 23, 2004, which is a continuation in part of Ser. No. 10/682,579filed Oct. 8, 2003, which claimed priority to Prov. No. 60/490,236 filedJul. 25, 2003 and Prov. No. 60/416,838 filed Oct. 8, 2002, all of whichare incorporated herein by reference in their entireties.

FIELD

The invention relates to the field of computers and telecommunications,and more particularly to an advanced processor for use in computers andtelecommunications applications.

BACKGROUND

Modern computers and telecommunications systems provide great benefitsincluding the ability to communicate information around the world.Conventional architectures for computers and telecommunicationsequipment include a large number of discrete circuits, which causesinefficiencies in both the processing capabilities and the communicationspeed.

For example, FIG. 1 illustrates such a conventional line card employinga number of discrete chips and technologies. In FIG. 1, conventionalline card 100 includes the following discrete components: Classification102, Traffic Manager 104, Buffer Memory 106, Security Co-Processor 108,Transmission Control Protocol (TCP)/Internet Protocol (IP) OffloadEngine 110, L3+Co-Processor 112, Physical Layer Device (PHY) 114, MediaAccess Control (MAC) 116, Packet Forwarding Engine 118, Fabric InterfaceChip 120, Control Processor 122, Dynamic Random-Access Memory (DRAM)124, Access Control List (ACL) Ternary Content-Addressable Memory (TCAM)126, and Multiprotocol Label Switching (MPLS) Static Random-AccessMemory (SRAM) 128. The card further includes Switch Fabric 130, whichmay connect with other cards and/or data.

Advances in processors and other components have improved the ability oftelecommunications equipment to process, manipulate, store, retrieve anddeliver information. Recently, engineers have begun to combine functionsinto integrated circuits to reduce the overall number of discreteintegrated circuits, while still performing the required functions atequal or better levels of performance. This combination has been spurredby the ability to increase the number of transistors on a chip with newtechnology and the desire to reduce costs. Some of these combinedintegrated circuits have become so highly functional that they are oftenreferred to as a System on a Chip (SoC). However, combining circuits andsystems on a chip can become very complex and pose a number ofengineering challenges. For example, hardware engineers want to ensureflexibility for future designs and software engineers want to ensurethat their software will run on the chip and future designs as well.

The demand for sophisticated new networking and communicationsapplications continues to grow in advanced switching and routing. Inaddition, solutions such as content-aware networking, highly integratedsecurity, and new forms of storage management are beginning to migrateinto flexible multi-service systems. Enabling technologies for these andother next generation solutions must provide intelligence and highperformance with the flexibility for rapid adaptation to new protocolsand services.

In order to take advantage of such high performance networking and dataprocessing capability, it is important that such systems be capable ofcommunicating with a variety of high bandwidth peripheral devices,preferably using a standardized, high-bandwidth bus. Although manyproprietary high-bandwidth buses are possible, using a standardized busallows the system to interface with a broader variety of peripherals,and thus enhances the overall value and utility of the system.

One high-bandwidth standardized bus that has become popular in recentyears is the PCI Express (PCI-E or PCIe) interface. The PCIe interface,originally proposed by Intel as a replacement for the very popular butbandwidth limited personal computer PCI interface, is both highbandwidth and, due to the fact that it has now become a standardcomponent on personal computer motherboards, very widely adopted.Hundreds or thousands of different peripherals are now available thatwork with the PCIe interface, making this interface particularly usefulfor the present advanced processing system.

In contrast to the earlier, parallel, PCI system, which encounteredbandwidth limitations due to problems with keeping the large number ofparallel circuit lines in synchronization with each other at high clockspeeds, the PCIe system is a very fast serial system. Serial systems useonly a very small limited number of circuit lines, typically two totransmit and two to receive, and this simpler scheme holds up better athigh clock speeds and high data rates. PCIe further increases bandwidthby allowing for multiple serial circuits. Depending upon the PCIeconfiguration, there can be as few as 1 bidirectional circuit, or asmany as 32 bidirectional serial circuits.

Although, on a hardware level, the serial PCIe system is radicallydifferent from the earlier parallel PCI system, the earlier PCI systemwas extremely successful, and the computer industry had made a massiveinvestment in earlier generation PCI hardware and software. To help makethe much higher bandwidth PCIe system compatible with the preexistingPCI hardware and software infrastructure, PCIe was designed to mimicmuch of the earlier parallel PCI data transport conventions. Earliergeneration software thus can continue to address PCIe devices as if theywere PCI devices, and the PCIe circuitry transforms the PCI data sendand receive requests into serial PCIe data packets, transmits orreceives these data packets, and then reassembles the serial PCIe datapackets back into a format that can be processed by software (andhardware) originally designed for the PCI format. The PCIe designintention of maintaining backward compatibility, while providing muchhigher bandwidth, has been successful and PCIe has now become a widelyused computer industry standard.

Although other workers, such as Stufflebeam (U.S. Pat. No. 7,058,738)have looked at certain issues regarding interfacing multiple CPUs tomultiple I/O devices through a single switch (such as a PCIe switch),this previous work has focused on less complex and typicallylower-performance multiple CPU configurations, that do not have tocontend with the issues that result when multiple cores must coordinatetheir activity via other high-speed (and often on-chip) communicationrings and interconnects.

Consequently, what is needed is an advanced processor that can takeadvantage of the new technologies while also providing high performancefunctionality. Additionally, this technology would be especially helpfulit included flexible modification ability, such as the ability tointerface with multiple high-bandwidth peripheral devices, usinghigh-bandwidth star topology buses such as the PCIe bus.

SUMMARY

The present invention provides useful novel structures and techniquesfor overcoming the identified limitations, and provides an advancedprocessor that can take advantage of new technologies while alsoproviding high performance functionality with flexible modificationability. The invention employs an advanced architecture System on a Chip(SoC) including modular components and communication structures toprovide a high performance device.

This advanced processor comprises a plurality of multithreaded processorcores each having a data cache and instruction cache. A data switchinterconnect (DSI) is coupled to each of the processor cores by itsrespective data cache, and configured to pass information among theprocessor cores. A level 2 (L2) cache, a memory bridge, and/or a supermemory bridge can also be coupled to the data switch interconnect (DSI)and configured to store information accessible to the processor cores.

A messaging network is coupled to each of the processor cores by thecore's respective instruction caches (message station). A plurality ofcommunication ports are connected to the messaging network. In oneaspect of the invention, the advanced telecommunications processorfurther comprises an interface switch interconnect (ISI) coupled to themessaging network and the plurality of communication ports andconfigured to pass information among the messaging network and thecommunication ports. This interface switch interconnect may alsocommunicate with the super memory bridge. The super memory bridge mayalso communicate with one or more communication ports and the previouslydiscussed DSI.

In the embodiment of the invention disclosed here, the messaging networkand the ISI connect to a PCI express (PCIe) interface, enabling theprocessor to interface with a broad variety of high-bandwidth PCIeperipherals.

Advantages of the PCIe embodiment of the invention include the abilityto provide high bandwidth communications between computer systems and alarge number of peripherals in an efficient, flexible, andcost-effective manner.

BRIEF DESCRIPTION OF THE FIGURES

The invention is described with reference to the FIGS, in which:

FIG. 1 illustrates a conventional line card;

FIG. 2A illustrates an exemplary advanced processor according to anembodiment of the invention, showing how the PCIe interface connects tothe processor;

FIG. 2B illustrates an exemplary advanced processor according to analternate embodiment of the invention, again showing how the PCIeinterface connects to the processor;

FIG. 3A illustrates a conventional single-thread single-issueprocessing;

FIG. 3B illustrates a conventional simple multithreaded scheduling;

FIG. 3C illustrates a conventional simple multithreaded scheduling witha stalled thread;

FIG. 3D illustrates an eager round-robin scheduling according to anembodiment of the invention;

FIG. 3E illustrates a multithreaded fixed-cycle scheduling according toan embodiment of the invention;

FIG. 3F illustrates a multithreaded fixed-cycle with eager round-robinscheduling according to an embodiment of the invention;

FIG. 3G illustrates a core with associated interface units according toan embodiment of the invention;

FIG. 3H illustrates an example pipeline of the processor according toembodiments of the invention;

FIG. 3I illustrates a core interrupt flow operation within a processoraccording to an embodiment of the invention;

FIG. 3J illustrates a programmable interrupt controller (PIC) operationaccording to an embodiment of the invention;

FIG. 3K illustrates a return address stack (RAS) operation for multiplethread allocation according to an embodiment of the invention;

FIG. 4A illustrates a data switch interconnect (DSI) ring arrangementaccording to an embodiment of the invention;

FIG. 4B illustrates a DSI ring component according to an embodiment ofthe invention;

FIG. 4C illustrates a flow diagram of an example data retrieval in theDSI according to an embodiment of the invention;

FIG. 5A illustrates a fast messaging ring component according to anembodiment of the invention;

FIG. 5B illustrates a message data structure for the system of FIG. 5A;

FIG. 5C illustrates a conceptual view of various agents attached to thefast messaging network (FMN) according to an embodiment of theinvention;

FIG. 5D illustrates network traffic in a conventional processing system;

FIG. 5E illustrates packet flow according to an embodiment of theinvention;

FIG. 5F shows a detailed view of how the PCIe interface connect theFMN/ISI and the PCIe I/O bus.

FIG. 5G shows an overview of the data fields of the 64 bit word messagessent between the FMN/ISI and the PCIe interface.

FIG. 5H shows how messages from the processor are translated by the PCIeinterface DMA into PCIe TLP requests, and how these various PCIe TLPpackets are reassembled by the PCIe interface DMA.

FIG. 5I shows a flow chart showing the asymmetry in acknowledgementmessages between PCIe read and write requests.

FIG. 6A illustrates a packet distribution engine (PDE) distributingpackets evenly over four threads according to an embodiment of theinvention;

FIG. 6B illustrates a PDE distributing packets using a round-robinscheme according to an embodiment of the invention;

FIG. 6C illustrates a packet ordering device (POD) placement duringpacket lifecycle according to an embodiment of the invention;

FIG. 6D illustrates a POD outbound distribution according to anembodiment of the invention.

DETAILED DESCRIPTION

The invention is described with reference to specific architectures andprotocols. Those skilled in the art will recognize that the descriptionis for illustration and to provide the best mode of practicing theinvention. The description is not meant to be limiting and references totelecommunications and other applications may be equally applicable togeneral computer applications, for example, server applications,distributed shared memory applications and so on. As described herein,reference is made to Ethernet Protocol, Internet Protocol, HyperTransport Protocol and other protocols, but the invention may beapplicable to other protocols as well. Moreover, reference is made tochips that contain integrated circuits while other hybrid ormeta-circuits combining those described in chip form is anticipated.Additionally, reference is made to an exemplary MIPS architecture andinstruction set, but other architectures and instruction sets can beused in the invention. Other architectures and instruction sets include,for example, x86, PowerPC, ARM and others.

A. ARCHITECTURE

The architecture is focused on a system designed to consolidate a numberof the functions performed on the conventional line card and to enhancethe line card functionality. In one embodiment, the invention is anintegrated circuit that includes circuitry for performing many discretefunctions. The integrated circuit design is tailored for communicationprocessing. Accordingly, the processor design emphasizes memoryintensive operations rather than computationally intensive operations.The processor design includes an internal network configured forrelieving the processor of burdensome memory access processes, which aredelegated to other entities for separate processing. The result is ahigh efficient memory access and threaded processing.

Again, one embodiment of the invention is designed to consolidate anumber of the functions performed on the conventional line card of FIG.1, and to enhance the line card functionality. In one embodiment, theinvention is an integrated circuit that includes circuitry forperforming many discrete functions. The integrated circuit design istailored for communication processing. Accordingly, the processor designemphasizes memory intensive operations rather than computationallyintensive operations. The processor design includes an internal networkconfigured for high efficient memory access and threaded processing asdescribed below.

FIG. 2A illustrates an exemplary advanced processor (200) according toan embodiment of the invention. The advanced processor is an integratedcircuit that can perform many of the functions previously tasked tospecific integrated circuits. For example, the advanced processorincludes a packet forwarding engine, a level 3 co-processor and acontrol processor. The processor can include other components, asdesired. As shown herein, given the number of exemplary functionalcomponents, the power dissipation is approximately 20 watts in theexemplary embodiment. Of course, in other embodiments of the invention,the power dissipation may be more or less than about 20 watts.

The exemplary processor is designed as a network on a chip. Thisdistributed processing architecture allows components to communicatewith one another and not necessarily share a common clock rate. Forexample, one processor component could be clocked at a relatively highrate while another processor component is clocked at a relatively lowrate. The network architecture further supports the ability to add othercomponents in future designs by simply adding the component to thenetwork. For example, if a future communication interface is desired,that interface can be laid out on the processor chip and coupled to theprocessor network. Then, future processors can be fabricated with thenew communication interface.

The design philosophy is to create a processor that can be programmedusing general purpose software tools and reusable components. Severalexemplary features that support this design philosophy include: staticgate design; low-risk custom memory design; flip-flop based design;design-for-testability including a full scan, memory built-in self-test(BIST), architecture redundancy and tester support features; reducedpower consumption including clock gating; logic gating and memorybanking; datapath and control separation including intelligently guidedplacement; and rapid feedback of physical implementation.

The software philosophy is to enable utilization of industry standarddevelopment tools and environment. The desire is to program theprocessing using general purpose software tools and reusable components.The industry standard tools and environment include familiar tools, suchas gcc/gdb and the ability to develop in an environment chosen by thecustomer or programmer.

The desire is also to protect existing and future code investment byproviding a hardware abstraction layer (HAL) definition. This enablesrelatively easy porting of existing applications and code compatibilitywith future chip generations.

Turning to the CPU core, the core is designed to be MIPS64 compliant andhave a frequency target in the range of approximately 1.5 GHz+.Additional exemplary features supporting the architecture include: 4-waymultithreaded single issue 10-stage pipeline; real time processingsupport including cache line locking and vectored interrupt support; 32KB 4-way set associative instruction cache; 32 KB 4-way set associativedata cache; and 128-entry translation-lookaside buffer (TLB).

One of the important aspects of the exemplary embodiment is thehigh-speed processor input/output (I/O), which is supported by: twoXGMII/SPI-4 (e.g., boxes 228 a and 228 b of FIG. 2A); three 1 Gb MACs;one 16-bit HyperTransport (e.g., box 232) that can scale to 800/1600 MHzmemory, including one Flash portion (e.g., box 226 of FIG. 2A) and twoQuad Data Rate (QDR2)/Double Data Rate (DDR2) SRAM portions; two 64-bitDDR2 channels that can scale to 400/800 MHz; and communication portsincluding PCIe (PCI-expanded or Expanded Peripheral ComponentInterconnect) ports (e.g., box 234 of FIG. 2A), Joint Test Access Group(JTAG) and Universal Asynchronous Receiver/Transmitter (UART) (e.g., box226).

The PCIe Communication Port:

Use of high bandwidth star topology serial communication buses, such asthe PCIe bus, is useful because it helps expand the power and utility ofthe processor. Before discussing how PCIe technology may be integratedinto this type of processor, a more detailed review of PCIe technologyis warranted.

As previously discussed, the PCIe bus is composed of one or more(usually multiple) high-bandwidth bidirectional serial connections. Eachbidirectional serial connection is called a “lane”. These serialconnections are in turn controlled by a PCIe switch that can createmultiple point-to-point serial connections between various PCIeperipherals (devices) and the PCIe switch in a star topologyconfiguration. As a result, each device gets its own direct serialconnection to the switch, and doesn't have to share this connection withother devices. This topology, along with the higher inherent speed ofthe serial connection, helps increase the bandwidth of the PCIe bus. Tofurther increase bandwidth, a PCIe device can connect to the switch withbetween 1 and 32 lanes. Thus a PCIe device that needs higher bandwidthcan utilize more PCIe lanes, and a PCIe device that needs lowerbandwidth can utilize fewer PCIe lanes.

Note that the present teaching of efficient means to interface serialbuses to this type of advanced processor is not limited to the PCIeprotocol per-se. As will be discussed, these techniques and methods canbe adapted to work with a broad variety of different star topologyhigh-speed serial bus protocols, including HyperTransport, InfiniBand,RapidIO, and StarFabric. To simplify discussion, however, PCIe will beused throughout this disclosure as a specific example.

As previously discussed, in order to maximize backward compatibilitywith the preexisting PCI parallel bus, PCIe designers elected to makethe new serial aspects of the PCIe bus as transparent (unnoticeable) tosystem hardware and software as possible. They did this by making thehigh level interface of the PCIe bus resemble the earlier PCI bus, andput the serial data management functions at a lower level, so thatpreexisting hardware and software would not have to contend with thevery different PCIe packet based serial data exchange format. Thus PCIedata packets send a wide variety of different signals, including controlsignals and interrupt signals.

Detailed information on the PCI express can be found in the book “PCIexpress system architecture” by Budruk, Anderson, and Shanley,Mindeshare, Inc. (2004), Addison Wesley.

Briefly, the PCIe protocol consists of the Physical layer (containing alogical sublayer and an electrical sublayer), the transaction layer, anda data link layer.

The physical layer (sometimes called the PIPE or PHY) controls theactual serial lines connecting various PCIe devices. It allows devicesto form one or more serial bidirectional “lanes” with the central PCIeswitch, and utilizes each device's specific hardware and bandwidth needsto determine exactly how many lanes should be allocated to the device.Since all communication is by the serial links, other messages, such asinterrupts and control signals are also sent by serial data packetsthrough these lanes. Instead of using clock pulses to synchronize data(which use a lot of bandwidth), the data packets are instead sent usinga sequential 8-bit/10-bit encoding scheme which in itself carries enoughclock information to ensure that the devices do not lose track of whereone byte begins and another byte ends. If data is being sent throughmultiple lanes to a device, this data will usually be interleaved withsequential bytes being sent on different lanes, which further increasesspeed. PCIe speeds are typically in the 2.5 Gigabit/second ranges, withfaster devices planned for the near future.

The transaction layer manages exactly what sort of traffic is movingover the serial lanes at any given moment of time. It utilizescredit-based flow control. PCIe devices signals (get credit for) anyextra receive buffers they may have. Whenever a sending device sends atransaction layer packet (here termed a PCIe-TLP to distinguish thisfrom a different “thread level parallelism” TLP acronym) to a receivingdevice, it, the sending device deducts a credit from this account, thusensuring that the buffer capability of the receiving device is notexceeded. When the receiving device has processed the data, it sends asignal back to the sending device signaling that it has free buffersagain. This way, a fair number of PCIe-TLP's can be reliably sentwithout cluttering bandwidth with a lot of return handshaking signalsfor each PCIe-TLP.

The data link layer handles the transaction layer packets. In order todetect any PCIe-TLP send or receive errors, the data link error bundlesthe PCIe-TLP with a 32 bit CRC checksum. If, for some reason, a givenPCIe-TLP fails checksum verification, this failure is communicated backto the originating device as a NAK using a separate type of data linklayer packet (DLLP). The originating device can then retransmit thePCIe-TLP.

Further details of how the PCIe bus can be integrated with the advancedprocessor will be given later in this discussion.

In addition to the PCIe bus, the interface may contain many other typesof devices. Also included as part of the interface are two Reduced GM II(RGM II) (e.g., boxes 230 a and 230 b of FIG. 2A) parts. Further,Security Acceleration Engine (SAE) (e.g., box 238 of FIG. 2A) can usehardware-based acceleration for security functions, such as encryption,decryption, authentication, and key generation. Such features can helpsoftware deliver high performance security applications, such as IPSecand SSL.

The architecture philosophy for the CPU is to optimize for thread levelparallelism (TLP) rather than instruction level parallelism (ILP)including networking workloads benefit from TLP architectures, andkeeping it small.

The architecture allows for many CPU instantiations on a single chip,which in turn supports scalability. In general, super-scalar designshave minimal performance gains on memory bound problems. An aggressivebranch prediction is typically unnecessary for this type of processorapplication and can even be wasteful.

The exemplary embodiment employs narrow pipelines because they typicallyhave better frequency scalability. Consequently, memory latency is notas much of an issue as it would be in other types of processors, and infact, any memory latencies can effectively be hidden by themultithreading, as described below.

Embodiments of the invention can optimize the memory subsystem withnon-blocking loads, memory reordering at the CPU interface, and specialinstruction for semaphores and memory barriers.

In one aspect of the invention, the processor can acquire and releasesemantics added to load/stores. In another aspect of embodiments of theinvention, the processor can employ special atomic incrementing fortimer support.

As described above, the multithreaded CPUs offer benefits overconventional techniques. An exemplary embodiment of the inventionemploys fine grained multithreading that can switch threads every clockand has 4 threads available for issue.

The multithreading aspect provides for the following advantages: usageof empty cycles caused by long latency operations; optimized for areaversus performance trade-off; ideal for memory bound applications;enable optimal utilization of memory bandwidth; memory subsystem; cachecoherency using MOSI (Modified, Own, Shared, Invalid) protocol; full mapcache directory including reduced snoop bandwidth and increasedscalability over broadcast snoop approach; large on-chip shared dualbanked 2 MB L2 cache; error checking and correcting (ECC) protectedcaches and memory; 2 64-bit 400/800 DDR2 channels (e.g., 12.8 GByte/speak bandwidth) security pipeline; support of on-chip standard securityfunctions (e.g., AES, DES/3DES, SHA-1, MD5, and RSA); allowance of thechaining of functions (e.g, encrypt->sign) to reduce memory accesses; 4Gbs of bandwidth per security-pipeline, excluding RSA; on-chip switchinterconnect; message passing mechanism for intra-chip communication;point-to-point connection between super-blocks to provide increasedscalability over a shared bus approach; 16 byte full-duplex links fordata messaging (e.g., 32 GB/s of bandwidth per link at 1 GHz); andcredit-based flow control mechanism.

Some of the benefits of the multithreading technique used with themultiple processor cores include memory latency tolerance and faulttolerance.

FIG. 2B illustrates an exemplary advanced processor according to analternate embodiment of the invention. This embodiment is provided toshow that the architecture can be modified to accommodate othercomponents, for example, video processor 215. In such a case, the videoprocessor can communicate with the processor cores, communicationnetworks (e.g. DSI and Messaging Network) and other components.

B. PROCESSOR CORES AND MULTI-THREADING

The exemplary advanced processor 200 of FIG. 2A includes a plurality ofmultithreaded processor cores 210 a-h. Each exemplary core includes anassociated data cache 212 a-h and instruction cache 214 a-h. Data SwitchInterconnect (DSI) 216 may be coupled to each of the processor cores 210a-h and configured to pass data among the processor cores and betweenthe L2 cache 208 and memory bridges 206, 208 for main memory access.Additionally, a messaging network 222 may be coupled to each of theprocessor cores 210 a-h and a plurality of communication ports 240 a-f.While eight cores are depicted in FIG. 2A, a lesser or greater number ofcores can be used in the invention. Likewise, in aspects of theinvention, the cores can execute different software programs androutines, and even run different operating systems. The ability to rundifferent software programs and operating systems on different coreswithin a single unified platform can be particularly useful where legacysoftware is desired to be run on one or more of the cores under an olderoperating system, and newer software is desired to be run on one or moreother cores under a different operating system or systems. Similarly, asthe exemplary processor permits multiple separate functions to becombined within a unified platform, the ability to run multipledifferent software and operating systems on the cores means that thedisparate software associated with the separate functions being combinedcan continue to be utilized.

The exemplary processor includes the multiple CPU cores 210 a-h capableof multithreaded operation. In the exemplary embodiment, there are eight4-way multithreaded MIPS64-compatible CPUs, which are often referred toas processor cores. Embodiments of the invention can include 32 hardwarecontexts and the CPU cores may operate at over approximately 1.5 GHz.One aspect of the invention is the redundancy and fault tolerant natureof multiple CPU cores. So, for example, if one of the cores failed, theother cores would continue operation and the system would experienceonly slightly degraded overall performance. In one embodiment, a ninthprocessor core may be added to the architecture to ensure with a highdegree of certainty that eight cores are functional.

The multithreaded core approach can allow software to more effectivelyuse parallelism that is inherent in many packet processing applications.Most conventional processors use a single-issue, single-threadedarchitecture, but this has performance limitations in typical networkingapplications. In aspects of the invention, the multiple threads canexecute different software programs and routines, and even run differentoperating systems. This ability, similar to that described above withrespect to the cores, to run different software programs and operatingsystems on different threads within a single unified platform can beparticularly useful where legacy software is desired to be run on one ormore of the threads under an older operating system, and newer softwareis desired to be run on one or more other threads under a differentoperating system or systems. Similarly, as the exemplary processorpermits multiple separate functions to be combined within a unifiedplatform, the ability to run multiple different software and operatingsystems on the threads means that the disparate software associated withthe separate functions being combined can continue to be utilized.Discussed below are some techniques used by the invention to improveperformance in single and multithreaded applications.

Referring now to FIG. 3A, a conventional single-thread single-issueprocessing is shown and indicated by the general reference character300A. The cycle numbers-are shown across the top of the blocks. “A”within the blocks can represent a first packet and “B” within the blockscan represent a next packet. The sub-numbers within the blocks canrepresent packet instructions and/or segments. The wasted cycles 5-10after a cache miss, as shown, result from no other instructions beingready for execution. The system must essentially stall to accommodatethe inherent memory latency and this is not desirable.

For many processors, performance is improved by executing moreinstructions per cycle, thus providing for instruction level parallelism(ILP). In this approach, more functional units are added to thearchitecture in order to execute multiple instructions per cycle. Thisapproach is also known as a single-threaded, multiple-issue processordesign. While offering some improvement over single-issue designs,performance typically continues to suffer due to the high-latency natureof packet processing applications in general. In particular,long-latency memory references usually result in similar inefficiencyand increased overall capacity loss.

As an alternate approach, a multithreaded, single-issue architecture maybe used. This approach takes advantage of, and more fully exploits, thepacket level parallelism commonly found in networking applications. Inshort, memory latencies can be effectively hidden by an appropriatelydesigned multithreaded processor. Accordingly, in such a threadeddesign, when one thread becomes inactive while waiting for memory datato return, the other threads can continue to process instructions. Thiscan maximize processor use by minimizing wasted cycles experienced byother simple multi-issue processors.

Referring now to FIG. 3B, a conventional simple multithreaded schedulingis shown and indicated by the general reference character 300B.Instruction Scheduler (IS) 302B can receive four threads: A, B, C, andD, as shown in the boxes to the left of IS 302B. Each cycle can simplyselect a different packet instruction from each of the threads in“round-robin” fashion, as shown. This approach generally works well aslong as every thread has an instruction available for issue. However,such a “regular” instruction issue pattern cannot typically be sustainedin actual networking applications. Common factors, such as instructioncache miss, data cache miss, data use interlock, and non-availability ofa hardware resource can stall the pipeline.

Referring now to FIG. 3C, a conventional simple multithreaded schedulingwith a stalled thread is shown and indicated by the general referencecharacter 300C. Instruction Scheduler (IS) 302C can receive fourthreads: A, B, and C, plus an empty “D” thread. As shown, conventionalround-robin scheduling results in wasted cycles 4, 8, and 12, thepositions where instructions from the D thread would fall if available.In this example, the pipeline efficiency loss is 25% during the timeperiod illustrated. An improvement over this approach that is designedto overcome such efficiency losses is the “eager” round-robin schedulingscheme.

Referring now to FIG. 3D, an eager round-robin scheduling according toan embodiment of the invention is shown and indicated by the generalreference character 300D. The threads and available instructions shownare the same as illustrated in FIG. 3C. However, in FIG. 3D, the threadscan be received by an Eager Round-Robin Scheduler (ERRS) 302D. The eagerround-robin scheme can keep the pipeline full by issuing instructionsfrom each thread in sequence as long as instructions are available forprocessing. When one thread is “sleeping” and does not issue aninstruction, the scheduler can issue an instruction from the remainingthree threads at a rate of one every three clock cycles, for example.Similarly, if two threads are inactive, the scheduler can issue aninstruction from the two active threads at the rate of one every otherclock cycle. A key advantage of this approach is the ability to rungeneral applications, such as those not able to take full advantage of4-way multithreading, at full speed. Other suitable approaches includemultithreaded fixed-cycle scheduling.

Referring now to FIG. 3E, an exemplary multithreaded fixed-cyclescheduling is shown and indicated by the general reference character300E. Instruction Scheduler (IS) 302E can receive instructions from fouractive threads: A, B, C, and D, as shown. In this programmablefixed-cycle scheduling, a fixed number of cycles can be provided to agiven thread before switching to another thread. In the exampleillustrated, thread A issues 256 instructions, which may be the maximumallowed in the system, before any instructions are issued from thread B.Once thread B is started, it may issue 200 instructions before handingoff the pipeline to thread C, and so on.

Referring now to FIG. 3F, an exemplary multithreaded fixed-cycle witheager round-robin scheduling is shown and indicated by the generalreference character 300F. Instruction Scheduler (IS) 302F can receiveinstructions from four active threads: A, B, C, and D, as shown. Thisapproach may be used in order to maximize pipeline efficiency when astall condition is encountered. For example, if thread A encounters astall (e.g., a cache miss) before it has issued 256 instructions, theother threads may be used in a round-robin manner to “fill up” thepotentially wasted cycles. In the example shown in FIG. 3F, a stallcondition may occur while accessing the instructions for thread A aftercycle 7, at which point the scheduler can switch to thread B for cycle8. Similarly, another stall condition may occur while accessing theinstructions for thread B after cycle 13, so the scheduler can thenswitch to thread C for cycle 14. In this example, no stalls occur duringthe accessing of instructions for thread C, so scheduling for thread Ccan continue though the programmed limit for the thread (e.g., 200), sothat the last C thread instruction can be placed in the pipeline incycle 214.

Referring now to FIG. 3G, a core with associated interface unitsaccording to an embodiment of the invention is shown and indicated bythe general reference character 300G. Core 302G can include InstructionFetch Unit (IFU) 304G, Instruction Cache Unit (ICU) 306G, Decouplingbuffer 308G, Memory. Management Unit (MMU) 310G, Instruction ExecutionUnit (IEU) 312G, and Load/Store Unit (LSU) 314. IFU 304G can interfacewith ICU 306G and IEU 312G can interface with LSU 314. ICU 306G can alsointerface with Switch Block (SWB)/Level 2 (L2) cache block 316G. LSU314G, which can be a Level 1 (L1) data cache, can also interface withSWB/L2 316G. IEU 312G can interface with Message (MSG) Block 318G and,which can also interface with SWB 320G. Further, Register 322G for usein accordance with embodiments can include thread ID (TID), programcounter (PC), and data fields.

According to embodiments of the invention, each MIPS architecture coremay have a single physical pipeline, but may be configured to supportmulti-threading functions (i.e., four “virtual” cores). In a networkingapplication, unlike a regular computational type of instruction scheme,threads are more likely to be waited on for memory accesses or otherlong latency operations. Thus, the scheduling approaches as discussedherein can be used to improve the overall efficiency of the system.

Referring now to FIG. 3H, an exemplary 10-stage (i.e., cycle) processorpipeline is shown and indicated by the general reference character 300H.In general operation, each instruction can proceed down the pipeline andmay take 10-cycles or stages to execute. However, at any given point intime, there can be up to 10 different instructions populating eachstage. Accordingly, the throughput for this example pipeline can be 1instruction completing every cycle.

Viewing FIGS. 3G and 3H together, cycles 1-4 may represent the operationof IFU 304G, for example. In FIG. 3H, stage or cycle 1 (IPG stage) caninclude scheduling an instruction from among the different threads(Thread Scheduling 302H). Such thread scheduling can includeround-robin, weighted round-robin (WRR), or eager round-robin, forexample. Further, an Instruction Pointer (IP) may be generated in theIPG stage. An instruction fetch out of ICU 306G can occur in stages 2(FET) and 3 (FE2) and can be initiated in Instruction Fetch Start 304Hin stage 2. In stage 3, Branch Prediction 306H and/or Return AddressStack (RAS) (Jump Register) 310H can be initiated and may complete instage 4 (DEC). Also in stage 4, the fetched instruction can be returned(Instruction Return 308H). Next, instruction as well as other relatedinformation can be passed onto stage 5 and also put in Decoupling buffer308G.

Stages 5-10 of the example pipeline operation of FIG. 3H can representthe operation of IEU 312G. In stage 5 (REG), the instruction may bedecoded and any required register lookup (Register Lookup 314H)completed. Also in stage 5, hazard detection logic (LD-Use Hazard 316H)can determine whether a stall is needed. If a stall is needed, thehazard detection logic can send a signal to Decouple buffer 308G toreplay the instruction (e.g., Decoupling/Replay 312H). However, if nosuch replay is signaled, the instruction may instead be taken out ofDecoupling buffer 308G. Further, in some situations, such as where ahazard/dependency is due to a pending long-latency operation (e.g., adata-cache miss), the thread may not be replayed, but rather put tosleep. In stage 6 (EXE), the instruction may be “executed,” which may,for example, include an ALU/Shift and/or other operations (e.g.,ALU/Shift/OP 318H). In stage 7 (MEM), a data memory operation can beinitiated and an outcome of the branch can be resolved (BranchResolution 320H). Further, the data memory lookup may extend to spanstages 7, 8 (RTN), and 9 (RT2), and the load data can be returned (LoadReturn 322H) by stage 9 (RT2). In stage 10 (WRB), the instruction-can becommitted or retired and all associated registers can be finally updated for the particular instruction.

In general, the architecture is designed such that there are no stallsin the pipeline. This approach was taken for both ease of implementationas well as increased frequency of operation. However, there are somesituations where a pipeline stall or stop is required. In suchsituations, Decoupling buffer 308G, which can be considered a functionalpart of IFU 304G, can allow for a restart or “replay” from a stop pointinstead of having to flush the entire pipeline and start the thread overto effect the stall. A signal can be provided by IFU 304G to Decouplingbuffer 308G to indicate that a stall is needed, for example. In oneembodiment, Decoupling buffer 308G can act as a queue for instructionswhereby each instruction obtained by IFU 304G also goes to Decouplingbuffer 308G. In such a queue, instructions may be scheduled out of orderbased on the particular thread scheduling, as discussed above. In theevent of a signal to Decoupling buffer 308G that a stall is requested,those instructions after the “stop” point can be re-threaded. On theother hand, if no stall is requested, instructions can simply be takenout of the decoupling buffer and the pipeline continued. Accordingly,without a stall, Decoupling buffer 308G can behave essentially like afirst-in first-out (FIFO) buffer. However, if one of several threadsrequests a stall, the others can proceed through the buffer and they maynot be held up.

As another aspect of embodiments of the invention, atranslation-lookaside buffer (TLB) can be managed as part of a memorymanagement unit (MMU), such as MMU 310G of FIG. 3G. This can includeseparate, as well as common, TLB allocation across multiple threads. The128-entry TLB can include a 64-entry joint main TLB and two 32-entrymicroTLBs, one each for the instruction and the data side. When atranslation cannot be satisfied by accessing the relevant microTLB, arequest may be sent to the main TLB. An interrupt or trap may occur ifthe main TLB also does not contain the desired entry.

In order to maintain compliance with the MIPS architecture, the main TLBcan support paired entries (e.g., a pair of consecutive virtual pagesmapped to different physical pages), variable page sizes (e.g., 4K to256M), and software management via TLB read/write instructions. Tosupport multiple threads, entries in the microTLB and in the main TLBmay be tagged with the thread ID (TID) of the thread that installedthem. Further, the main TLB can be operated in at least two modes. In a“partitioned” mode, each active thread may be allocated an exclusivesubset or portion of the main TLB to install entries and, duringtranslation, each thread only sees entries belonging to itself. In“global” mode, any thread may allocate entries in any portion of themain TLB and all entries may be visible to all threads. A “de-map”mechanism can be used during main TLB writes to ensure that overlappingtranslations are not introduced by different threads.

Entries in each microTLB can be allocated using a not-recently-used(NRU) algorithm, as one example. Regardless of the mode, threads mayallocate entries in any part of the microTLB. However, translation inthe microTLB may be affected by mode. In global mode, all microTLBentries may be visible to all threads, but in partitioned mode, eachthread may only see its own entries. Further, because the main TLB cansupport a maximum of one translation per cycle, an arbitration mechanismmay be used to ensure that microTLB “miss” requests from all threads areserviced fairly.

In a standard MIPS architecture, unmapped regions of the address spacefollow the convention that the physical address equals the virtualaddress. However, according to embodiments of the invention, thisrestriction is lifted and unmapped regions can undergovirtual-to-physical mappings through the microTLB/mainTLB hierarchywhile operating in a “virtual-MIPS” mode. This approach allows a user toisolate unmapped regions of different threads from one another. As abyproduct of this approach, however, the normal MIPS convention thatmainTLB entries containing an unmapped address in their virtual pagenumber (VPN2) field can be considered invalid is violated. In oneembodiment of the invention, this capability can be restored to the userwhereby each entry in the mainTLB can include a special “master valid”bit that may only be visible to the user in the virtual MIPS-mode. Forexample, an invalid entry can be denoted by a master valid bit value of“0” and a valid entry can be denoted by a master valid bit value of “1.”

As another aspect of the invention, the system can support out-of-orderload/store scheduling in an in-order pipeline. As an exampleimplementation, there can be a user-programmable relaxed memory orderingmodel so as to maximize overall performance. In one embodiment, theordering can be changed by user programming to go from a stronglyordered model to a weakly ordered model. The system can support fourtypes: (i) Load-Load Re-ordering; (ii) Load-Store Re-ordering; (ii)Store-Store Re-ordering; and (iv) Store-Load Re-ordering. Each type ofordering can be independently relaxed by way of a bit vector in aregister. If each type is set to the relaxed state, a weakly orderedmodel can be achieved.

Referring now to FIG. 3I, a core interrupt flow operation within aprocessor according to an embodiment of the invention is shown andindicated by the general reference character 300I. ProgrammableInterrupt Controller (PIC), as will be discussed in more detail belowwith reference to FIG. 3J, may provide an interrupt including InterruptCounter and MSG Block to Accumulates 302I. Accordingly, operation 300Ican occur within any of the processors or cores of the overall system.Functional block Schedules Thread 304I can receive control interfacefrom block 302I. Extensions to the MIPS architecture can be realized byshadow mappings that can include Cause 306I to EIRR 308I as well asStatus 310I to EIMR 312I. The MIPS architecture generally only provides2 bits for software interrupts and 6 bits for hardware interrupts foreach of designated status and cause registers. This MIPS instructionarchitecture compatibility can be retained while providing extensions,according to embodiments of the invention.

As shown in more detail in FIG. 3I, a shadow mapping for Cause 306I toEIRR 308I for an interrupt pending can include bits 8-15 of the Cause306I register mapping to bits 0-7 of EIRR 308I. Also, a softwareinterrupt can remain within a core, as opposed to going through the PIC,and can be enacted by writing to bits 8 and/or 9 of Cause 306I. Theremaining 6 bits of Cause 306I can be used for hardware interrupts.Similarly, a shadow mapping for Status 310I to EIMR 312I for a mask caninclude bits 8-15 of the Status 310I register mapping to bits 0-7 ofEIMR 312I. Further, a software interrupt can be enacted by writing tobits 8 and/or 9 of Status 310I while the remaining 6 bits can be usedfor hardware interrupts. In this fashion, the register extensionsaccording to embodiments of the invention can provide much moreflexibility in dealing with interrupts. In one embodiment, interruptscan also be conveyed via the non-shadowed bits 8-63 of EIRR 308I and/orbits 8-63 of EIMR 312I.

Referring now to FIG. 3J, a PIC operation according to an embodiment ofthe invention is shown and indicated by the general reference character300J. For example, flow 300J may be included in an implementation of box226 of FIG. 2A. In FIG. 3J, Sync 302J can receive an interruptindication and provide a control input to Pending 304J control block.Pending 304J, which can effectively act as an interrupt gateway, canalso receive system timer and watch dog timer indications. ScheduleInterrupt 306J can receive an input from Pending 304J. InterruptRedirection Table (IRT) 308J can receive an input from ScheduleInterrupt 306J.

Each interrupt and/or entry of IRT 308J can include associatedattributes (e.g., Attribute 314J) for the interrupt, as shown. Attribute314J can include CPU Mask 316-1J, Interrupt Vector 316-2J, as well asfields 316-3J and 316-4J, for examples. Interrupt Vector 316-2J can be a6-bit field that designates a priority for the interrupt. In oneembodiment, a lower number in Interrupt Vector 316-2J can indicate ahigher priority for the associated interrupt via a mapping to EIPR 308I,as discussed above with reference to FIG. 3I. In FIG. 3J, Scheduleacross CPU & Threads 310J can receive an input from block 308J, such asinformation from Attribute 314J. In particular, CPU Mask 316-1J may beused to indicate to which of the CPUs or cores the interrupt is to bedelivered. Delivery 312J block can receive an input from block 310J.

In addition to the PIC, each of 32 threads, for example, may contain a64-bit interrupt vector. The PIC may receive interrupts or requests fromagents and then deliver them to the appropriate thread. As one exampleimplementation, this control may be software programmable. Accordingly,software control may elect to redirect all external type interrupts toone or more threads by programming the appropriate PIC controlregisters. Similarly, the PIC may receive an interrupt event orindication from the PCI-X interface (e.g., PCI-X 234 of FIG. 2A), whichmay in turn be redirected to a specific thread of a processor core.Further, an interrupt redirection table (e.g., IRT 308J of FIG. 3J) maydescribe the identification of events (e.g., an interrupt indication)received by the PIC as well as information related to its direction toone or more “agents.” The events can be redirected to a specific core byusing a core mask, which can be set by software to specify the vectornumber that may be used to deliver the event to a designated recipient.An advantage of this approach is that it allows the software to identifythe source of the interrupt without polling.

In the case where multiple recipients are programmed for a given eventor interrupt, the PIC scheduler can be programmed to use a global“round-robin” scheme or a per-interrupt-based local round-robin schemefor event delivery. For example, if threads 5, 14, and 27 are programmedto receive external interrupts, the PIC scheduler may deliver the firstexternal interrupt to thread 5, the next one to thread 14, the next oneto thread 27, then return to thread 5 for the next interrupt, and so on.

In addition, the PIC also may allow any thread to interrupt any otherthread (i.e., an inter-thread interrupt). This can be supported byperforming a store (i.e., a write operation) to the PIC address space.The value that can be used for such a write operation can specify theinterrupt vector and the target thread to be used by the PIC for theinter-thread interrupt. Software control can then use standardconventions to identify the inter-thread interrupts. As one example, avector range may be reserved for this purpose.

As discussed above with reference to FIGS. 3G and 3H, each core caninclude a pipeline decoupling buffer (e.g., Decoupling 308G of FIG. 3G).In one aspect of embodiments of the invention, resource usage in anin-order pipeline with multiple threads can be maximized. Accordingly,the decoupling buffer is “thread aware” in that threads not requesting astall can be allowed to flow through without stopping. In this fashion,the pipeline decoupling buffer can re-order previously scheduledthreads. As discussed above, the thread scheduling can only occur at thebeginning of a pipeline. Of course, re-ordering of instructions within agiven thread is not normally performed by the decoupling buffer, butrather independent threads can incur no penalty because they can beallowed to effectively bypass the decoupling buffer while a stalledthread is held-up.

In one embodiment of the invention, a 3-cycle cache can be used in thecore implementation. Such a 3-cycle cache can be an “off-the-shelf” celllibrary cache, as opposed to a specially-designed cache, in order toreduce system costs. As a result, there may be a gap of three cyclesbetween the load and the use of a piece of data and/or an instruction.The decoupling buffer can effectively operate in and take advantage ofthis 3-cycle delay. For example, if there was only a single thread, a3-cycle latency would be incurred. However, where four threads areaccommodated, intervening slots can be taken up by the other threads.Further, branch prediction can also be supported. For branches correctlypredicted, but not taken, there is no penalty. For branches correctlypredicted and taken, there is a one-cycle “bubble” or penalty. For amissed prediction, there is a 5-cycle bubble, but such a penalty can bevastly reduced where four threads are operating because the bubbles cansimply be taken up by the other threads. For example, instead of a5-cycle bubble, each of the four threads can take up one so that only asingle bubble penalty effectively remains.

As discussed above with reference to FIGS. 3D, 3E, and 3F, instructionscheduling schemes according to embodiments of the invention can includeeager round-robin scheduling (ERRS), fixed number of cycles per thread,and multithreaded fixed-cycle with ERRS. Further, the particularmechanism for activating threads in the presence of conflicts caninclude the use of a scoreboard mechanism, which can track long latencyoperations, such as memory access, multiply, and/or divide operations.

Referring now to FIG. 3K, a return address stack (RAS) operation formultiple thread allocation is shown and indicated by the generalreference character 300K. This operation can be implemented in IFU 304Gof FIG. 3G and as also indicated in operation 310H of FIG. 3H, forexample. Among the instructions supported in embodiments of theinvention are: (i) a branch instruction where a prediction is whether itis taken or not taken and the target is known; (ii) a jump instructionwhere it is always taken and the target is known; and (iii) a jumpregister where it is always taken and the target is retrieved from aregister and/or a stack having unknown contents.

In the example operation of FIG. 3K, a Jump-And-Link (JAL) instructioncan be encountered (302K) to initiate the operation. In response to theJAL, the program counter (PC) can be placed on the return address stack(RAS)(304K). An example RAS is shown as Stack 312K and, in oneembodiment, Stack 312K is a first-in last-out (FILO) type of stack toaccommodate nested subroutine calls. Substantially in parallel withplacing the PC on Stack 312K, a subroutine call can be made (306K).Various operations associated with the subroutine instructions can thenoccur (308K). Once the subroutine flow is complete, the return addresscan be retrieved from Stack 312K (310K) and the main program cancontinue (316K) following any branch delay (314K).

For multiple thread operation, Stack 312K can be partitioned so thatentries are dynamically configured across a number of threads. Thepartitions can change to accommodate the number of active threads.Accordingly, if only one thread is in use, the entire set of entriesallocated for Stack 312K can be used for that thread. However, ifmultiple threads are active, the entries of Stack 312K can bedynamically configured to accommodate the threads so as to utilize theavailable space of Stack 312K efficiently.

In a conventional multiprocessor environment, interrupts are typicallygiven to different CPUs for processing on a round-robin basis or bydesignation of a particular CPU for the handling of interrupts. However,in accordance with embodiments of the invention, PIC 226 of FIG. 2A,with operation shown in more detail in FIG. 3J, may have the ability toload balance and redirect interrupts across multiple CPUs/cores andthreads in a multithreaded machine. As discussed above with reference toFIG. 3J, IRT 308J can include attributes for each interrupt, as shown inAttribute 314J. CPU Mask 316-1J can be used to facilitate load balancingby allowing for certain CPUs and/or threads to be masked out of theinterrupt handling. In one embodiment, CPU Mask may be 32-bits wide toallow for any combination of 8 cores, each having 4 threads, to bemasked. As an example, Core-2 210 c and Core-7 210 h of FIG. 2A may beintended to be high availability processors, so CPU Mask 316-1 J of FIG.3J may have its corresponding bits set to “1” for each interrupt in IRT308J so as to disallow any interrupt processing on Core-2 or Core-7.

Further, for both CPUs/cores as well as threads, a round-robin scheme(e.g., by way of a pointer) can be employed among those cores and/orthreads that are not masked for a particular interrupt. In this fashion,maximum programmable flexibility is allowed for interrupt loadbalancing. Accordingly, operation 300J of FIG. 3J allows for two levelsof interrupt scheduling: (i) the scheduling of 306J, as discussed above;and (ii) the load balancing approach including CPU/core and threadmasking.

As another aspect of embodiments of the invention, thread-to-threadinterrupting is allowed whereby one thread can interrupt another thread.Such thread-to-thread interrupting may be used for synchronization ofdifferent threads, as is common for telecommunications applications.Also, such thread-to-thread interrupting may not go through anyscheduling according to embodiments of the invention.

C. DATA SWITCH AND L2 CACHE

Returning now to FIG. 2A, the exemplary processor may further include anumber of components that promote high performance, including: an 8-wayset associative on-chip level-2 (L2) cache (2 MB); a cache coherentHyper Transport interface (768 Gbps); hardware acceleratedQuality-of-Service (QOS) and classification; security hardwareacceleration—AES, DES/3DES, SHA-1, MD5, and RSA; packet orderingsupport; string processing support; TOE hardware (TCP Offload Engine);and numerous IO signals. In one aspect of an embodiment of theinvention, data switch interconnect 216 may be coupled to each of theprocessor cores 210 a-h by its respective data cache 212 a-h. Also, themessaging network 222 may be coupled to each of the processor cores 210a-h by its respective instruction cache 214 a-h. Further, in one aspectof an embodiment of the invention, the advanced telecommunicationsprocessor can also include an L2 cache 208 coupled to the data switchinterconnect and configured to store information accessible to theprocessor cores 210 a-h. In the exemplary embodiment, the L2 cacheincludes the same number of sections (sometimes referred to as banks) asthe number of processor cores. This example is described with referenceto FIG. 4A, but it is also possible to use more or fewer L2 cachesections.

As previously discussed, embodiments of the invention may include themaintenance of cache coherency using MOSI (Modified, Own, Shared,Invalid) protocol. The addition of the “Own” state enhances the “MSI”protocol by allowing the sharing of dirty cache lines across processcores. In particular, an example embodiment of the invention may presenta fully coherent view of the memory to software that may be running onup to 32 hardware contexts of 8 processor cores as well as the I/Odevices. The MOSI protocol may be used throughout the L1 and L2 cache(e.g., 212 a-h and 208, respectively, of FIG. 2A) hierarchy. Further,all external references (e.g., those initiated by an I/O device) maysnoop the L1 and L2 caches to ensure coherency and consistency of data.In one embodiment, as will be discussed in more detail below, aring-based approach may be used to implement cache coherency in amultiprocessing system. In general, only one “node” may be the owner fora piece of data in order to maintain coherency.

According to one aspect of embodiments of the invention, an L2 cache(e.g., cache 208 of FIG. 2A) may be a 2 MB, 8-way set-associativeunified (i.e., instruction and data) cache with a 32B line size.Further, up to 8 simultaneous references can be accepted by the L2 cacheper cycle. The L2 arrays may run at about half the rate of the coreclock, but the arrays can be pipelined to allow a request to be acceptedby all banks every core clock with a latency of about 2 core clocksthrough the arrays. Also, the L2 cache design can be “non-inclusive” ofthe L1 caches so that the overall memory capacity can be effectivelyincreased.

As to ECC protection for an L2 cache implementation, both cache data andcache tag arrays can be protected by SECDED (Single Error CorrectionDouble Error Detection) error protecting codes. Accordingly, all singlebit errors are corrected without software intervention. Also, whenuncorrectable errors are detected, they can be passed to the software ascode-error exceptions whenever the cache line is modified. In oneembodiment, as will be discussed in more detail below, each L2 cache mayact like any other “agent” on a ring of components.

According to another aspect of embodiments of the invention, “bridges”on a data movement ring may be used for optimal redirection of memoryand I/O traffic. Super Memory I/O Bridge 206 and Memory Bridge 218 ofFIG. 2A may be separate physical structures, but they may beconceptually the same. The bridges can be the main gatekeepers for mainmemory and I/O accesses, for example. Further, in one embodiment, theI/O can be memory-mapped.

Referring now to FIG. 4A, a data switch interconnect (DSI) ringarrangement according to an embodiment of the invention is shown andindicated by the general reference character 400A. Such a ringarrangement can be an implementation of DSI 216 along with Super MemoryI/O Bridge 206 and Memory Bridge 218 of FIG. 2A. In FIG. 4A, Bridge 206can allow an interface between memory & I/O and the rest of the ring.Ring elements 402 a-j each correspond to one of the cores 210 a-h andthe memory bridges of FIG. 2A. Accordingly, element 402 a interfaces toL2 cache L2 a and Core-0 210 a, and element 402 b interfaces to L2 b andCore 210 b, and so on through 402 h interfacing to L2 h and Core 210 h.Bridge 206 includes an element 402 i on the ring and bridge 218 includesan element 402 j on the ring.

As shown in FIG. 4A, four rings can make up the ring structure in anexample embodiment: Request Ring (RQ), Data Ring (DT), Snoop Ring (SNP),and Response Ring (RSP). The communication on the rings is packet basedcommunication. An exemplary RQ ring packet includes destination ID,transaction ID, address, request type (e.g., RD, RD.sub.13EX, WR, UPG),valid bit, cacheable indication, and a byte enable, for example. Anexemplary DT ring packet includes destination ID, transaction ID, data,status (e.g., error indication), and a valid bit, for example. Anexemplary SNP ring packet includes destination ID, valid bit, CPU snoopresponse (e.g., clean, shared, or dirty indication), L2 snoop response,bridge snoop response, retry (for each of CPU, bridge, and L2), AERR(e.g., illegal request, request parity), and a transaction ID, forexample. An exemplary RSP ring packet includes all the fields of SNP,but may represent a “final” status, as opposed to the “in-progress”status of the RSP ring.

Referring now to FIG. 4B, a DSI ring component according to anembodiment of the invention is shown and indicated by the generalreference character 400B. Ring component 402 b-0 may correspond to oneof the four rings RQ, DT, SNP, or RSP, in one embodiment. Similarly,ring components 402 b-1, 402 b-2, and 402 b-3 may each correspond to oneof the four rings. As an example, a “node” can be formed by thesummation of ring components 402 b-0, 402 b-1, 402 b-2, and 402 b-3.

Incoming data or “Ring In” can be received in flip-flop 404B. An outputof flip-flop 404B can connect to flip-flops 406B and 408B as sell asmultiplexer 416B. Outputs of flip-flops 406B and 408B can be used forlocal data use. Flip-flop 410B can receive an input from the associatedL2 cache while flip-flop 412B can receive an input from the associatedCPU. Outputs from flip-flops 410B and 412B can connect to multiplexer414B. An output of multiplexer 414B can connect to multiplexer 416B andan output of multiplexer 416B can connect to outgoing data or “RingOut.” Also, ring component 402 b-0 can receive a valid bit signal.

Generally, higher priority data received on Ring In will be selected bymultiplexer 416B if the data is valid (e.g., Valid Bit=“1”). If not, thedata can be selected from either the L2 or the CPU via multiplexer 414B.Further, in this example, if data received on Ring In is intended forthe local node, flip-flops 406B and/or 408B can pass the data onto thelocal core instead of allowing the data to pass all the way around thering before receiving it again.

Referring now to FIG. 4C, a flow diagram of an example data retrieval inthe DSI according to an embodiment of the invention is shown andindicated by the general reference character 400C. The flow can begin inStart 452 and a request can be placed on the request ring (RQ) (454).Each CPU and L2 in the ring structure can check for the requested data(456). Also, the request can be received in each memory bridge attachedto the ring (458). If any CPU or L2 has the requested data (460), thedata can be put on the data ring (DT) by the node having the data (462).If no CPU or L2 has found the requested data (460), the data can beretrieved by one of the memory bridges (464). An acknowledgement can beplaced on the snoop ring (SNP) and/or the response ring (RSP) by eitherthe node that found the data or the memory bridge (466) and the flow cancomplete in End (468). In one embodiment, the acknowledgement by thememory bridge to the SNP and/or RSP ring may be implied.

In an alternative embodiment, the memory bridge would not have to waitfor an indication that the data has not been found in any of the L2caches in order to initiate the memory request. Rather, the memoryrequest (e.g., to DRAM), may be speculatively issued. In this approach,if the data is found prior to the response from the DRAM, the laterresponse can be discarded. The speculative DRAM accesses can help tomitigate the effects of the relatively long memory latencies.

D. MESSAGE PASSING NETWORK

Also in FIG. 2A, in one aspect of an embodiment of the invention, theadvanced telecommunications processor can include Interface SwitchInterconnect (ISI) 224 coupled to the messaging network 222 and a groupof communication ports 240 a-f, and configured to pass information amongthe messaging network 222 and the communication ports 240 a-f.

Referring now to FIG. 5A, a fast messaging ring component or stationaccording to an embodiment of the invention is shown and indicated bythe general reference character 500A. An associated ring structure mayaccommodate point-to-point messages as an extension of the MIPSarchitecture, for example. The “Ring In” signal can connect to bothInsertion Queue 502A and Receive Queue (RCVQ) 506A. The insertion queuecan also connect to multiplexer 504A, the output of which can be “RingOut.” The insertion queue always gets priority so that the ring does notget backed-up. Associated registers for the CPU core are shown in dashedboxes 520A and 522A. Within box 520A, buffers RCV Buffer 510A-0 throughRCV Buffer 510A-N can interface with RCVQ 506A. A second input tomultiplexer 504A can connect to Transmit Queue (XMTQ) 508A. Also withinbox 520A, buffers XMT Buffer 512A-0 through XMT Buffer 512A-N caninterface with XMTQ 508A. Status 514A registers can also be found in box520A. Within dashed box 522A, memory-mapped Configuration Registers 516Aand Credit Based Flow Control 518A can be found.

Referring now to FIG. 5B, a message data structure for the system ofFIG. 5A is shown and indicated by the general reference character 500B.Identification fields may include Thread 502B, Source 504B, andDestination 508B. Also, there can be a message size indicator Size 508B.The identification fields and the message size indicator can formSideboard 514B. The message or data to be sent itself (e.g., MSG 512B)can include several portions, such as 510B-0, 510B-1, 510B-2, and510B-3. According to embodiments, the messages may be atomic so that thefull message cannot be interrupted.

The credit-based flow control can provide a mechanism for managingmessage sending, for example. In one embodiment, the total number ofcredits assigned to all transmitters for a target/receiver cannot exceedthe sum of the number of entries in its receive queue (e.g., RCVQ 506Aof FIG. 5A). For example, 256 may be the total number of credits in oneembodiment because the size of the RCVQ of each target/receiver may be256 entries. Generally, software may control the assignment of credits.At boot-up time, for example, each sender/xmitter or participating agentmay be assigned some default number of credits. Software may then befree to allocate credits on a per-transmitter basis. For example, eachsender/xmitter can have a programmable number of credits set by softwarefor each of the other targets/receivers in the system. However, not allagents in the system may be required to participate as targets/receiversin the distribution of the transmit credits. In one embodiment, Core-0credits can be programmed for each one of Core-1, Core-2, . . . Core-7,RGM II.sub.130, RGMII.sub.131, XGM II/SPI-4.2.sub.—0,XGMII/SPI-4.2.sub.—1, POD0, POD1, . . . POD4, etc. The Table 1 belowshows an example distribution of credits for Core-0 as a receiver:

TABLE 1 Transmit Agents Allowed Credits (Total of 256) Core-0 0 Core-132 Core-2 32 Core-3 32 Core-4 0 Core-5 32 Core-6 32 Core-7 32 POD0 32RGMII_0 32 All Others 0

In this example, when Core-1 sends a message of size 2 (e.g., 2 64-bitdata elements) to Core-0, the Core-1 credit in Core-0 can be decrementedby 2 (e.g., from 32 to 30). When Core-0 receives a message, the messagecan go into the RCVQ of Core-0. Once the message is removed from theRCVQ of Core-0, that message storage space may essentially be freed-upor made available. Core-0 can then send a signal to the sender (e.g., afree credit signal to Core-1) to indicate the amount of space (e.g., 2)additionally available. If Core-1 continues to send messages to Core-0without corresponding free credit signals from Core-0, eventually thenumber of credits for Core-1 can go to zero and Core-1 may not be ableto send any more messages to Core-0. Only when Core-0 responds with freecredit signals could Core-1 send additional messages to Core-0, forexample.

Referring now to FIG. 5C, a conceptual view of how various agents may beattached to the fast messaging network (FMN) according to an embodimentof the invention is shown and indicated by the general referencecharacter 500C. The eight cores (Core-0 502C-0 through Core-7 502C-7)along with associated data caches (D-cache 504C-0 through 504C-7) andinstruction caches (I-cache 506C-0 through 506C-7) can interface to theFMN. Further, Network I/O Interface Groups can also interface to theFMN. Associated with Port A, DMA 508C-A, Parser/Classifier 512C-A, andXGMII/SPI-4.2 Port A 514C-A can interface to the FMN through PacketDistribution Engine (PDE) 510C-A. Similarly, for Port B, DMA 508C-B,Parser/Classifier 512C-B, and XGMII/SPI-4.2 Port B 514C-B can interfaceto the FMN through PDE 510C-B. Also, DMA 516C, Parser/Classifier 520C,RGMII Port A 522C-A, RGMII Port B 522C-B, RGMII Port C 522C-C, RGMIIPort D 522C-D can interface to the FMN through PDE 518C. Also, SecurityAcceleration Engine 524C including DMA 526C and DMA Engine 528C caninterface to the FMN. PCIe bus 534 can also interface to the FMN and/orthe interface switch interconnect (ISI). This interface is shown in moredetail in FIGS. 5F through 5I.

As an aspect of embodiments of the invention, all agents (e.g.,cores/threads or networking interfaces, such as shown in FIG. 5C) on theFMN can send a message to any other agent on the FMN. This structure canallow for fast packet movement among the agents, but software can alterthe use of the messaging system for any other appropriate purpose by sodefining the syntax and semantics of the message container. In anyevent, each agent on the FMN includes a transmit queue (e.g., 508A) anda receive queue (e.g., 506A), as discussed above with reference to FIG.5A. Accordingly, messages intended for a particular agent can be droppedinto the associated receive queue. All messages originating from aparticular agent can be entered into the associated transmit queue andsubsequently pushed on the FMN for delivery to the intended recipient.

In another aspect of embodiments of the invention, all threads of thecore (e.g., Core-0 502C-0 through Core-7 502C-7 or FIG. 5C) can sharethe queue resources. In order to ensure fairness in sending outmessages, a “round-robin” scheme can be implemented for acceptingmessages into the transmit queue. This can guarantee that all threadshave the ability to send out messages even when one of them is issuingmessages at a faster rate. Accordingly, it is possible that a giventransmit queue may be full at the time a message is issued. In such acase, all threads can be allowed to queue up one message each inside thecore until the transmit queue has room to accept more messages. As shownin FIG. 5C, the networking interfaces use the PDE to distribute incomingpackets to the designated threads. Further, outgoing packets for thenetworking interfaces can be routed through packet ordering software.

Referring now to FIG. 5D, network traffic in a conventional processingsystem is shown and indicated by the general reference character 500D.The Packet Input can be received by Packet Distribution 502D and sentfor Packet Processing (504D-0 through 504D-3). Packet Sorting/Ordering506D can receive the outputs from Packet Processing and can providePacket Output. While such packet-level parallel-processing architecturesare inherently suited for networking applications, but an effectivearchitecture must provide efficient support for incoming packetdistribution and outgoing packet sorting/ordering to maximize theadvantages of parallel packet processing. As shown in FIG. 5D, everypacket must go through a single distribution (e.g., 502D) and a singlesorting/ordering (e.g., 506D). Both of these operations have aserializing effect on the packet stream so that the overall performanceof the system is determined by the slower of these two functions.

Referring now to FIG. 5E, a packet flow according to an embodiment ofthe invention is shown and indicated by the general reference character500E. This approach provides an extensive (i.e., scalable)high-performance architecture enabling flow of packets through thesystem. Networking Input 502E can include RGMII, XGMII, and/or SPI-4.2and/or PCIe interface configured ports. After the packets are received,they can be distributed via Packet Distribution Engine (PDE) 504E usingthe Fast Messaging Network (FMN) to one of the threads for PacketProcessing 506E: Thread 0, 1, 2, and so on through Thread 31, forexample. The selected thread can perform one or more functions asprogrammed by the packet header or the payload and then the packet on toPacket Ordering Software 508E. As an alternative embodiment, a PacketOrdering Device (POD), as shown in box 236 of FIG. 2A, for example, maybe used in place of 508E of FIG. 5E. In either implementation, thisfunction sets up the packet ordering and then passes it onto theoutgoing network (e.g., Networking Output 510E) via the FMN. Similar tothe networking input, the outgoing port can be any one of the configuredRGMII, XGMII, or SPI-4.2 interfaces or PCIe bus, for example.

E. INTERFACE SWITCH

In one aspect of embodiments of the invention, the FMN can interface toeach CPU/core, as shown in FIG. 2A. Such FMN-to-core interfacing mayinclude push/pop instructions, waiting for a message instruction, andinterrupting on a message arrival. In the conventional MIPSarchitecture, a co-processor or “COP2” space is allocated. However,according to embodiments of the invention, the space designated for COP2is instead reserved for messaging use via the FMN. In one embodiment,software executable instructions may include message send (MsgSnd),message load (MsgLd), message-to-COP2(MTC2), message-from-COP2(MFC2),and message wait (MsgWait). The MsgSnd and MsgLd instructions caninclude target information as well as message size indications. The MTC2and MFC2 instructions can include data transfers from/to localconfiguration registers, such as Status 514A and registers 522A of FIG.5A. The MsgWait instruction can include the operation of essentiallyentering a “sleep” state until a message is available (e.g.,interrupting on message arrival).

As another aspect of embodiments of the invention, fast messaging (FMN)ring components can be organized into “buckets.” For, example, RCVQ 506Aand XMTQ 508A of FIG. 5A may each be partitioned across multiple bucketsin similar fashion to the thread concept, as discussed above.

In one aspect of embodiments of the invention, a Packet DistributionEngine (PDE) can include each of the XGMII/SPI-4.2 interfaces and fourRGMII interfaces and PCIe interfaces to enable efficient andload-balanced distribution of incoming packets to the processingthreads. Hardware accelerated packet distribution is important for highthroughput networking applications. Without the PDE, packet distributionmay be handled by software, for example. However, for 64B packets, onlyabout 20 ns is available for execution of this function on an XGMII typeinterface. Further, queue pointer management would have to be handleddue to the single-producer multiple-consumer situation. Such asoftware-only solution is simply not able to keep up with the requiredpacket delivery rate without impacting the performance of the overallsystem.

According to an embodiment of the invention, the PDE can utilize theFast Messaging Network (FMN) to quickly distribute packets to thethreads designated by software as processing threads. In one embodiment,the PDE can implement a weighted round-robin scheme for distributingpackets among the intended recipients. In one implementation, a packetis not actually moved, but rather gets written to memory as thenetworking interface receives it. The PDE can insert a “PacketDescriptor” in the message and then send it to one of the recipients, asdesignated by software. This can also mean that not all threads mustparticipate in receiving packets from any given interface.

PCIe Interface

Referring now to FIG. 5F, 5F shows a close up of the interface betweenthe Fast Messaging Network (FMN) and interface switch interconnect (ISI)(540) and the PCIe interface (534) previously shown in FIG. 2A (234),and FIG. 5C (534). The fast messaging network and/or the interfaceswitch interconnect (540) sends various signals to the PCIe interface toboth control the interface, and send and receive data from theinterface.

To speed development of such devices, it will usually be advantageous torely upon pre-designed PCIe components as much as possible. SuitablePCIe digital cores, physical layer (PHY), and verification componentsmay be obtained from different vendors. Typically these components willbe purchased as intellectual property and integrated circuit designpackages, and these design packages used, in conjunction with customizedDMA design software, to design integrated circuit chips capable ofinterfacing to fast message buses of advanced processors, and in turninterfacing with PCIe bus hardware.

One important design consideration is to streamline (simplify) the PCIeinterface and commands as much as possible. This streamlining processkeeps the interface both relatively simple and relatively fast, andallows single chip multiple core processors, such as the presentinvention, to control PCIe devices with relatively small amounts ofsoftware and hardware overhead.

As will be discussed, the present design utilizes the Fast messagingnetwork (FMN)/interface switch interconnect in combination with acustomized DMA engine (541) embedded as part of the PCIe interface unit(534). The DMA engine essentially serves as a translator between thevery different memory storage protocols used by the processor cores andthe various PCIe devices. Offloading the translation process to acustomized DMA engine greatly reduces the computing demands on the coreprocessors, freeing them up for other tasks.

To briefly review, DMA (Direct Memory Access) circuits allow hardware toaccess memory independently of the processor core CPUs. The DMA acts tocopy memory chunks between devices. Although the CPU initiates thisprocess with an appropriate DMA command, the CPU can then do otherthings while the DMA executes the memory copy command.

In this embodiment, a customized DMA engine was constructed thataccepted the short, the highly optimized, processor FMN and ISI messagesas input. The DMA then both translated the FMN/ISI messages intoappropriate PCIe-TLP packets (which were then handled by the other PCIecircuitry in the interface), and then automatically handled the originalmemory copy command requested by the core processors.

Thus the DMA translated the memory copy request into PCIe-TLP packets,and sent it on to the other PCIe circuitry that sends the appropriatecommands to the appropriate PCIe device on the PCIe bus via PCIe-TLPpackets. After these packets had been sent by the PCIe circuitry, thedesignated PCIe device completes the task. After the designated PCIe busdevice has completed its assigned task, the PCIe bus device would thenreturn appropriate PCIe-TLP packets to the PCIe interface. The PCIeinterface DMA accepts these PCIe-TLP packets, and translates them intothe appropriate memory copy commands, and otherwise manages the taskoriginally assigned by the processor core without need for furtherattention from the processor cores. As a result, the processor and theprocessor cores can now communicate with a broad variety of differentPCIe devices with minimal processor core computational overhead.

The processor cores communicate with the PCIe interface via short (1-264 bit words) messages sent over the FMN/ISI. The PCIe interface isdesigned to accept incoming messages that are assigned to four differentbuckets. Each bucket can have a queue of up to 64 entries per bucket,for a maximum queue dept of 256 messages.

The PCIe interface is also designed to output 4 different classes ofmessages. These output messages can each be stored in a queue of up to 4messages per class, for a maximum output queue depth of 16 outputmessages.

The commands sent from the FMN/ISI to the PCIe interface are as follows:

Message in Buckets Name Bucket Description Link0 0 Message to readmemory data from I/O Bus Slave DMA interface and write memory dataacross the PCIe Link0 Load (also called DMA posted) Link0 1 Message toread memory data from the PCIe Link0 and DMA write memory data acrossthe I/O Bus Slave interface Store (also called DAM non-posted) Link1 2Message to read memory data from I/O Bus Slave DMA interface and writememory data across the PCIe Load link1 (also called DMA posted) Link1 3Message to read memory data from the PCIe link1 DMA and write memorydata across the I/O Bus Slave Store interface (also called DAMnon-posted)

Most of the communication from the processor cores to the PCIe interfaceis in the form of simple, terse, 2 64-bit word memory read/writecommands. The processor cores will normally assume that these commandshave been successfully completed. As a result, the PCIe interfacecommunicates back only terse OK, Not OK, status messages in the form ofa very terse single 64-bit word. The various message out classes areshown below:

Message Out Classes Name Class Description Link0 DMA Load 0 Returnmessage for Message in Bucket 0 Link0 DMA Store 1 Return Message forMessage in Bucket 1 Link1 DMA Load 2 Return Message for Message inBucket 2 Link1 DMA Store 3 Return Message for Message in Bucket 3

FIG. 5G shows an overview of the data fields in the two, terse, 64 bitwords sent by the processor cores via the FMN/ISI to the DMA engine onthe PCIe interface device, as well as an overview of the single, terse,64 bit-word sent from the DMA engine on the PCIe interface device backto the processor cores via the FMN/ISI. As can be seen, most of the wordspace is taken up with DMA source address, DMA destination address, andmemory data byte count information. The remainder of the two wordsconsists of various control information bits and fields, as shown below:

The fields of the FMN/ISI message to the PCIe interface DMA engine areas follows

Word Field Bits Description 0 Reserved 63:61 0 COHERENT 60 I/O Bus Slaverequest coherent enable 0 L2ALLOC 59 I/O Bus Slave request L2 Allocationenable 0 RDX 58 I/O Bus Slave request read exclusive enable 0 RETEN 57Return message enable 0 RETID 56:50 Return message target ID 0 TID 49:40Message tag ID 0 SRC 39:0 DMA source address 1 TD 63 PCIe request TDfield 1 EP 62 PCIe request EP field 1 ATTR 61:00 PCIe request Attr field1 BC 59:40 DMA data byte count 1 DEST 39:0 DMA destination address

Here, the COHERENT, RDX, RETEN, TD, EP ATTR fields can be considered tobe star topology serial bus control fields, or more specifically PCIebus control fields.

The outgoing return message is essentially a glorified ACK or OK orproblem message. It is composed of a single 64 bit word. Its fields areshown below:

Field Bits Description Reserved 63:14 FLUSH 13 Flush is enabled for themessage MSGERR 12 Incoming message error IOBERR 11 I/O Bus error PCIEERR10 PCIe link error TID 9:0 Message Tag ID

To simplify this discussion, the progress of these return messages willnot be discussed in detail, however some of this return path is shownindicated in FIG. 5F (555).

Returning to FIG. 5F, the incoming messages from the processor arestored in a message box (542) circuit, which can hold 4 buckets(different types) of messages with a buffer of 64 messages per type.

The messages are then processed through a round robin scheduler and amessage bus arbiter to the DMA engine (541). Memory load (write)instructions are processed by the DMA load posted requestor portion ofthe DMA engine (543), translated into appropriate PCIe-TCP data packets,and are then sent to the I/O bus slave circuit (545). This is then sentto the I/O bus arbiter circuit (546), which handles various PCI I/O busrequests using a round robin scheduler. The I/O bus arbiter then sendsthe packets to other PCIe circuitry, such as the I/O bus requestor (547)and the PCIe link completors (548). These work with the PCIe controlarbiter (549) to manage the actual PCIe physical layer circuitry (550)that does the actual hardware PCIe packet sending and receiving. ThePCIe-TCP packets then are put on the PCIe bus (551) and sent to thevarious PCIe devices (not shown). The write tag tracker tracks thestatus of reads out and content in.

In this embodiment, all write commands do not expect responses and arecomplete once the commands are accepted, however all read and readexclusive commands expect return responses.

In this embodiment, in order to simplify the design, it is assumed thatall I/O bus master read request completions can be returned out oforder. In this embodiment, it is also assumed that all requests are forcontiguous bytes, and that for the I/O, configuration, and registeraddress spaces, the most number of byes requested in a command is 4 andthe bytes do not cross the Double Word address boundary.

FIG. 5H shows an overview of the hardware flow of messages from theprocessor cores through the FMN to the PCIe interface device. Themessages will typically originate from the processor cores (560)(previously shown in FIG. 5 c (502C-0 to 502C-7), and be processed (561)for sending onto the FMN fast messaging network/interface switchinterconnect ISI (562) and then be received and processed by the initialstages of the PCIe interface (not shown). Eventually, after processingand appropriate round robin scheduling and arbitration, the writemessages are sent to the DMA load requestor portion of the DMA engine(563). The DMA load requestor then reads to the DRAM or cache (558) viaI/O interconnect (559), retrieves the appropriate data from DRAM orCache (558), and then does the appropriate translation and calculationsneeded to translate the data and request from the original DMA requestformat (shown in 5G) to appropriate (usually multiple) PCI-TLP0,PCI-TLP1 . . . PCI-TLPn (564) requests needed to access the requestedmemory locations on the relevant PCIe device. These requests are thenrouted through the other PCIe circuits to the PCIe physical layer (565),where they are then transmitted on the PCIe bus.

As previously discussed, to simplify the system and make processing timefaster, in this embodiment, the length of the write data plus thedestination address double word offset cannot exceed the maximum payloadlength of the PCIe-TLP. Further, the write request cannot cross 4K-byteaddress boundaries. The PCIe-TLPs are further divided into 32-bytesource addresses.

Return PCIe-TLP data packets: Memory read (store) requests (again thetwo 64 bit words shown in FIG. 5G) also travel through the FMN to thePCIe interface, where they are passed to the DMA store requestor in theDMA engine. The maximum length of the read request depends on theconfiguration's maximum payload size. When the source address of the DMAstore message is not Double Word aligned, the length of the first returnPCIe-TLP packets is 1, 2, or 3. (566). As before, in this embodiment,the read request PCIe-TLP packets cannot cross 4K byte source addressboundaries.

The DMA store requestor return packet handler (567) will make theappropriate writes via the I/O (559) to cache or dram (558), consult theoriginal DMA instructions, as well as the contents of the PCIe-TLPpackets, and either generate additional PCIe-TLP packets to forward thedata onto the appropriate PCIe device. It will then notify the processorcores (CPU) when read or write to the DRAM is complete.

The PCIe physical layer receiver receives (566) receives PCIe-TLPpackets (567) from various PCIe devices. The DMA store requestor enginedetermines if the appropriate way to respond to the original DMA commandis or is not to send out additional PCIe-TLP packets to other PCIedevices. If so, it sends appropriate PCIe-TMP commands out again. If theappropriate response to the received PCIe-TLP packets and the originalDMA command is to write the data to L2 cache or place it back on theFMN/ISI bus, the DMA will again make the appropriate decision and routeit to the FMN/ISI as needed.

FIG. 5I shows a flow chart of how messages flow from the processor coresto the PCIe interface, and shows the asymmetry in return messagegeneration between PCIe write and read requests. While write requests donot generate return confirmation messages, read messages do. After theprocessor (570) sends a PCIe message (571), this message is parsed bythe PCIe interface (571) and sent to the DMA engine (572). The DMAengine makes the appropriate reads and writes to DRAM or Cache via theinterconnect I/O (572A), and retrieves the needed data where it isretransmitted in the form of PCIe-TMP packets. If the requestedoperation (573) was a write (574), then the operation is completedsilently. If the requested operation was a read (575), then the systemgenerates a receive-complete acknowledgement (576), receives the readdata (577), and again makes the appropriate DRAM or Cache writes via theinterconnect I/O (577A), determines which processor or device requestedthe data (578), and places the data to the appropriate I/O interconnect(579) to return the data to the requesting device or processor. Oncethis is done (580), the PCIe will clear a tracking tag (581) that ismonitoring the progress of the transaction, indicating completion.

Referring now to FIG. 6A, a PDE distributing packets evenly over fourthreads according to an embodiment of the invention is shown andindicated by the general reference character 600A. In this example,software may choose threads 4 through 7 for possible reception ofpackets. The PDE can then select one of these threads in sequence todistribute each packet, for example. In FIG. 6A, Networking Input can bereceived by Packet Distribution Engine (PDE) 602A, which can select oneof Thread 4, 5, 6, or 7 for packet distribution. In this particularexample, Thread 4 can receive packet 1 at time t.sub.i and packet 5 attime t.sub.5, Thread 5 can receive packet 2 at time t.sub.2 and packet 6at time t.sub.6, Thread 6 can receive packet 3 at time t.sub.3 andpacket 7 at time t.sub.7, and Thread 7 can receive packet 4 at timet.sub.4 and packet 8 at time t.sub.8.

Referring now to FIG. 6B, a PDE distributing packets using a round-robinscheme according to an embodiment of the invention is shown andindicated by the general reference character 600B. As describe abovewith reference to the FMN, software can program the number of creditsallowed for all receivers from every transmitter. Since the PDE isessentially a transmitter, it can also use the credit information todistribute the packets in a “round-robin” fashion. In FIG. 6B, PDE 602Bcan receive Networking Input and provide packets to the designatedthreads (e.g., Thread 0 through Thread 3), as shown. In this example,Thread 2 (e.g., a receiver) may be processing packets more slowly thanthe other threads. PDE 602B can detect the slow pace of creditavailability from this receiver and adjust by guiding packets to themore efficiently processing threads. In particular, Thread 2 has theleast number of credits available within the PDE at cycle t.sub.11.Although the next logical receiver of packet 11 at cycle t.sub.11 mayhave been Thread 2, the PDE can identify a processing delay in thatthread and accordingly select Thread 3 as the optimal target fordistribution of packet 11. In this particular example, Thread 2 cancontinue to exhibit processing delays relative to the other threads, sothe PDE can avoid distribution to this thread. Also, in the event thatnone of the receivers has room to accept a new packet, the PDE canextend the packet queue to memory.

Because most networking applications are not very tolerant of the randomarrival order of packets, it is desirable to deliver packets in order.In addition, it can be difficult to combine features of parallelprocessing and packet ordering in a system. One approach is to leave theordering task to software, but it then becomes difficult to maintainline rate. Another option is to send all packets in a single flow to thesame processing thread so that the ordering is essentially automatic.However, this approach would require flow identification (i.e.,classification) prior to packet distribution and this reduces systemperformance. Another drawback is the throughput of the largest flow isdetermined by the performance of the single thread. This prevents singlelarge flows from sustaining their throughput as they traverse thesystem.

According to an embodiment of the invention, an advancedhardware-accelerated structure called a Packet Ordering Device (POD) canbe used. An objective of the POD is to provide an unrestricted use ofparallel processing threads by re-ordering the packets before they aresent to the networking output interface. Referring now to FIG. 6C, a PODplacement during packet lifecycle according to an embodiment of theinvention is shown and indicated by the general reference character600C. This figure essentially illustrates a logical placement of the PODduring the life cycle of the packets through the processor. In thisparticular example, PDE 602C can send packets to the threads, as shown.Thread 0 can receive packet 1 at time t.sub.1, packet 5 at time t.sub.5,and so on through cycle t.sub.n—3. Thread 1 can receive packet 2 at timet.sub.2, packet 6 at time t.sub.6, and so on through cycle t.sub.n—2.Thread 2 can receive packet 3 at time t.sub.3, packet 7 at time t.sub.7,and so on through time t.sub.n—1. Finally, Thread 3 can receive packet 4at time t.sub.4, packet 8 at time t.sub.8, and so on through timet.sub.n.

Packet Ordering Device (POD) 604C can be considered a packet sorter inreceiving the packets from the different threads and then sending toNetworking Output. All packets received by a given networking interfacecan be assigned a sequence number. This sequence number can then beforwarded to the working thread along with the rest of the packetinformation by the PDE. Once a thread has completed processing thepacket, it can forward the packet descriptor along with the originalsequence number to the POD. The POD can release these packets to theoutbound interface in an order strictly determined by the originalsequence numbers assigned by the receiving interface, for example.

In most applications, the POD will receive packets in a random orderbecause the packets are typically processed by threads in a randomorder. The POD can establish a queue based on the sequence numberassigned by the receiving interface and continue sorting packets asreceived. The POD can issue packets to a given outbound interface in theorder assigned by the receiving interface. Referring now to FIG. 6D, aPOD outbound distribution according to an embodiment of the invention isshown and indicated by the general reference character 600D. As can beseen in Packet Ordering Device (POD) 602D, packets 2 and 4 can beinitially sent to the POD by executing threads. After several cycles, athread can complete work on packet 3 and place it in the POD. Thepackets may not yet be ordered because packet 1 is not yet in place.Finally, packet 1 is completed in cycle t.sub.7 and placed in the PODaccordingly. Packets can now be ordered and the POD can begin issuingpackets in the order: 1, 2, 3, 4. If packet 5 is received next, it isissued in the output following packet 4. As the remaining packets arereceived, each can be stored in the queue (e.g., a 512-deep structure)until the next higher number packet is received. At such time, thepacket can be added to the outbound flow (e.g., Networking Output).

It is possible that the oldest packet may never arrive in the POD, thuscreating a transient head-of-line blocking situation. If not handledproperly, this error condition would cause the system to deadlock.However, according to an aspect of the embodiment, the POD is equippedwith a time-out mechanism designed to drop a non-arriving packet at thehead of the list once a time-out counter has expired. It is alsopossible that packets are input to the POD at a rate which fills thequeue capacity (e.g., 512 positions) before the time-out counter hasexpired. According to an aspect of the embodiment, when the POD reachesqueue capacity, the packet at the head of the list can be dropped and anew packet can be accepted. This action may also remove any head-of-lineblocking situation as well. Also, software may be aware that a certainsequence number will not be entered into the POD due to a bad packet, acontrol packet, or some other suitable reason. In such a case, softwarecontrol may insert a “dummy” descriptor in the POD to eliminate thetransient head-of-line blocking condition before allowing the POD toautomatically react.

According to embodiments of the invention, five programmable PODs may beavailable (e.g., on chip) and can be viewed as generic “sorting”structures. In one example configuration, software control (i.e., via auser) can assign four of the PODs to the four networking interfaceswhile retaining one POD for generic sorting purposes. Further, the PODscan simply be bypassed if so desired for applications wheresoftware-only control suffices.

F. MEMORY INTERFACE AND ACCESS

In one aspect of embodiments of the invention, the advancedtelecommunications processor can further include memory bridge 218coupled to the data switch interconnect and at least one communicationport (e.g., box 220), and configured to communicate with the data switchinterconnect and the communication port.

In one aspect of the invention, the advanced telecommunicationsprocessor can further include super memory bridge 206 coupled to thedata switch interconnect (DSI), the interface switch interconnect and atleast one communication port (e.g., box 202, box 204), and configured tocommunicate with the data switch interconnect, the interface switchinterconnect and the communication port.

In another aspect of embodiments of the invention, memory ordering canbe implemented on a ring-based data movement network, as discussed abovewith reference to FIGS. 4A, 4B, and 4C.

G. CONCLUSION

Advantages of the invention include the ability to provide highbandwidth communications between computer systems and memory in anefficient and cost-effective manner. In particular, this embodiment ofthe invention focuses on a novel PCIe interface that enhances thecapability of the advanced processor by enabling the processor to workwith a variety of different PCIe devices.

Having disclosed exemplary embodiments and the best mode, modificationsand variations may be made to the disclosed embodiments while remainingwithin the subject and spirit of the invention as defined by thefollowing Claims.

What is claimed is:
 1. A processor, comprising: at least two processorcores; at least two communication ports; and a messaging networkconfigured to: receive, from a first communication port from among theat least two communication ports, at least two packets of packet datafor respective processing by the at least two processor cores, transmit,to the at least two processor cores, the at least two packets of packetdata for the respective processing, receive, from the at least twoprocessor cores, at least two processed packets of packet data, andtransmit the at least two processed packets of packet data to a secondcommunication port from among the at least two communication ports. 2.The processor of claim 1, wherein the respective processing includesprocessing of the at least two packets of packet data by the at leasttwo processor cores based on information included in headers of the atleast two packets of packet data.
 3. The processor of claim 1, whereinthe respective processing includes processing of the at least twopackets of packet data by the at least two processor cores based onpayload information included in the at least two packets of packet data.4. The processor of claim 1, wherein the messaging network is configuredto receive the at least two packets of packet data in an order that isbased on a current number of credits associated with the at least twoprocessor cores.
 5. The processor of claim 4, wherein the current numberof credits associated with each of the at least two processor cores isbased on a processing speed associated with the at least two processorcores.
 6. The processor of claim 4, wherein the current number ofcredits associated with the at least two processor cores is based on adelay in processing of the packet data by the at least two processorcores.
 7. The processor of claim 5, wherein the messaging network isconfigured to transmit a packet, from among the at least two packets ofpacket data, to a processor core other than the at least two processorcores when a delay in processing of the packet data is determined to begreater than a predetermined threshold.
 8. The processor of claim 1,wherein the at least two packets of packet data are assigned at leasttwo sequence numbers which are forwarded to the at least two processorcores and included in the at least two processed packets of packet datato allow the at least two processed packets of packet data to be outputbased on the sequence number assigned to the at least two packets ofpacket data.
 9. The processor of claim 1, wherein at least one of the atleast two processor cores includes a data cache and an instructioncache, and the messaging network is disposed between the instructioncache of the at least one of the at least two processor cores and atleast one of the at least two communication ports.
 10. The processor ofclaim 9, further comprising: a data switch arrangement coupled to thedata cache of the at least one of the at least two processor cores, thedata switch arrangement being configured to transfer memory relatedinformation.
 11. A method, comprising: receiving, at a messaging networkfrom a first communication port from among at least two communicationports, at least two packets of packet data for respective processing bythe at least two processor cores, the messaging network being associatedwith the at least two processor cores and the at least two communicationports; transmitting, from the messaging network to the at least twoprocessor cores, the at least two packets of packet data for therespective processing, receiving, at the messaging network from the atleast two processor cores, at least two processed packets of packetdata, and transmitting, from the messaging network to a secondcommunication port from among the at least two communication ports, theat least two processed packets of packet data.
 12. The method of claim11, wherein the respective processing includes processing of the atleast two packets of packet data by the at least two processor coresbased on information included in headers of the at least two packets ofpacket data.
 13. The method of claim 11, wherein the respectiveprocessing includes processing of the at least two packets of packetdata by the at least two processor cores based on payload informationincluded in the at least two packets of packet data.
 14. The method ofclaim 11, wherein the receiving the at least two packets of packet dataincludes receiving the at least two packets of packet data in an orderthat is based on a current number of credits associated with the atleast two processor cores.
 15. The method of claim 14, wherein thecurrent number of credits associated with each of the at least twoprocessor cores is based on a processing speed of packet data by the atleast two processor cores.
 16. The method of claim 14, wherein thecurrent number of credits associated with the at least two processorcores is based on a delay in processing of the packet data by the atleast two processor cores.
 17. The method of claim 16, wherein thetransmitting the at least two packets of packet data for respectiveprocessing includes transmitting a packet, from among the at least twopackets of packet data, to a processor core other than the at least twoprocessor cores when the delay in processing of the packet data isdetermined to be greater than a predetermined threshold.
 18. The methodof claim 11, further comprising: assigning at least two sequence numbersto the at least two packets of packet data; forwarding the at least twosequence numbers to the at least two processor cores; including the atleast two sequence numbers in the at least two processed packets ofpacket data, and outputting the at least two processed packets of packetdata based on the sequence number assigned to each of the at least twopackets of packet data.
 19. A processor, comprising: at least twoprocessor cores, the at least two processor cores including at least twodata caches and at least two instruction caches; a data switcharrangement coupled to the at least two data cache of each of the atleast two processor cores, the data switch arrangement being configuredto transfer information among the at least two processor cores; and amessaging network configured to: receive, from a first communicationport, at least two packets of packet data for respective processing bythe at least two processor cores, transmit, to the at least twoprocessor cores, the at least two packets of packet data for therespective processing, receive, from the at least two processor cores,at least two processed packets of packet data, and transmit the at leasttwo processed packets of packet data to a second communication port. 20.The multi-core processor of claim 19, wherein each of the at least twopackets of packet data is assigned a sequence number which is forwardedto the at least two processor cores and is included in the at least twoprocessed packets of packet data to allow the at least two processedpackets of packet data to be output based on the sequence numberassigned to each of the at least two packets of packet data.